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ABSTRACT 

The evaluative implications of adult literacy for 
all, as delineated by the World Conference on Education for All (held 
in Jomtien, Thailand, on March 5-9, 1990), are examined. These 
implications are presented in four interrelated sets related to 
different stakeholders at different parts of the system. The 
Education for All (EFA) initiatives encompass progressive concepts of 
development and education. Education is seen as a fundamental human 
right, and basic education for all is recognized as a historical 
moral imperative. The structure of an overall evaluation of the EFA 
initiatives should have two important aspects: (1) meta-purposes of 
developing and reinforcing a culture of information and building 
institutional capacity; and (2) focus on combining institutional 
accreditation with program evaluation. The evaluation system should 
incorporate the following principles: internal evaluation; 
macro-evaluation and micro-evaluation; necsitsary and sufficient data; 
quantitative and qualitative evaluation; and evaluation that serves 
the facilitator, program designer, and policy maker- The case study 
of a national evaluation of Swedish schools and adult education 
reflects these principles as they must be applied in EFA evaluation, 
A 13-item list of references is included. (SLD) 
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EVALUATION IMPLICATIONS OF "EDUCATION FOR ALL": 



FOCUS ON ADULT LITERACY 



By 
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This paper was presented at the session on "Perspectives 
and Implications of Education for All^ II" held at the annual 
conference of the Comparative and International Education 
Society, Pittsburgh, March 14-17, 1991. 

The paper has benefitted greatly from the presentations, 
including one by the author, made earlier at the panel on 
"Evaluation Implications of the World Conference on Education 
for All" organized at Evaluation '90: Evaluation and the 
Formulation of Public Policy , the annual meeting of the American 
Evaluation Association, October 18^-20, 1990, Washington, D.C, 



EVALUATION IMPLICATIONS OF "EDUCATION FOR ALL": 



FOCUS ON ADULT LITERACY 
H.S. Bhola 

The declaration of the "World Conference on Education for 
All (WC-EFA) " (1) is at the same time a statement of development 
philosophy, educational policy, technical assistance strategy, 
plannincf, institution building and evaluation (The Inter-Agency 
Commission, 1990a, 1990b) . The implications of the evaluation 
agenda build into the EFA 

Declaration and Framework documents (Inter-Agency Commission, 
1990b) , therefore, go far beyond evaluation methodology and 
measurement techniques. 

It is important to note that the EFA initiative does not 
cover ail education, but only basic education. Basic education 
in this context covers two things: the universalization of 
primary education for children and youth, and the near- 
universalization of adult literacy which is broadly construed to 
include reading, functionality, and awareness. in this paper, we 
will focus on the evaluation implications of adult liceracy for 
all — or almost all. Problems arising from the need to 
interface the evaluation system for primary education with the 
evaluation system for adult literacy will be referred to if and 
when necessary. 



I 

EFA'S EVALUATION AGENDA 

A careful scanning of the Declaration (D) and the 
Framework (F) documents, delivers an impressive evaluation 
agenda (Inter-Agency Commission, 1990b) . 

Evaluation needs are seen as most comprehensive: "These 
needs comprise both essential learning tools (such as literacy, 
oral expression, numeracy, and problem solving) and the basic 
learning content (such as knowledge, skills, values, and 
attitudes) required by human beings to be able to survive, to 
develop their full capacities, to live and work in dignity, to 
participate fully in development, to improve quality of their 
lives, to make informed decisions, and to continue learnina 
(p. D-3)." 
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In evaluating both tool skills, and learning content, 
emphasis will be on results — "actual learning acquisition, and 
outcome, rather than exclusively upon enrolment, continued 
participation in organized programmes and completion of 
certification requirements (p. D-5) . Program effectiveness will 
be assessed in terms of behavioral changes and impact. 

Indicators of achievements, outputs and impact will be 
devised and standards of acceptable performance will be revised 
when appropriate. Both quantities and qualities will be 
reported. There will be the need to "specify expected 
attainments and outcomes in reference to terminal performance 

specifications within an appropriate time-frame formulated 

in terms such that progress toward them can be observed and 
measured (p. F-2)." 

Lockheed (1990) , while not formally representing the 
World Bank, does communicate the World Bank view quite well in 
regard to the Bank's expectations about evaluation. Also, while 
her frame of reference is formal education, her remarks apply 
equally well to out-of -school education. she informs that the 
Bank wishes to make program decisions "grounded on evidence of 
their impact on student learning." Immediacy and timeliness are 
essential. Therefore, instruments are needed that will be 
amenable to "immediate analysis". 

Realizing that there is a paucity of trained personnel in 
the area of evaluation in the Third World, the Bank wishes to 
assist the Third World professionals to: 

"(i) develop educational assessment-tests capable of monitoring 
achievement t ends over time; (ii) design and implement school- 
based sample surveys that will provide information about student 
achievement and the schooling context; and (iii) organize, 
combine and analyze the assessment-test, school survey and data 
extracted from existing data bases to provide a more complete 
picture of how schools and educational policies affect learning 
(Lockheed, 1990:4) 

The context data that the Bank has in mind for developing 
complete pictures of schooling are "socio-economic 
characteristics of students, fiscal resources, teachers, 
schools, curricula, classroom instruction and student 
participation." (Lockheed, 1990:10) 

Clearly, the expectations are very high. 
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IMPLICATIONS OF THE EVALUATION AGENDA: 
IMMEDIATE AND FAR-REACHING 

The rest of this paper is structured in three parts: 
First, the implications of the evaluation agenda of EFA are 
explicated and arranged in interrelated sets going from the 
immediate implications to f ar^-reaching implications. Second, 
the contradictions and conflicts inherent in the total set of 
these implications are discussed. Finally, suggestions are made 
as to how a System of evaluation might be structured that will 
reconcile at least some of the contradictions while accepting 
some of the inevitable anomalies of this less than perfect world. 

INTERRELATED SETS OF IMPLICATIONS 

The evaluation implications of adult literacy are 
presented in the following schema. Each set of implications is 
interrelated with a subsequent set of implications at a deeper 
level of the system. The implications we have listed relate to 
different stakeholders located at various verticle and horizontal 
points of the system and, therefore, arise from different vantage 
points. In terms of the values embedded in the contemporary 
theory of development and education, some of the implications can 
be seen as positive, and others as negative. 



\/ 

First Set 

That projects, programs, and campaigns of adult literacy 
world wide, must test for achievements, outputs and impact, and 
not merely report €?nrolments and ratios of participation. 

That all literacy projects, programs and campaigns should 
have clear program objectives in regard to both literacy skills 
and learning content. 

That there should be a careful development gj indicators 
and systematic norm setting in relation to program goals and of 
standards of quantity and quality to judge achievements. 

That each literacy initiative should establish a 
management information system (MIS) to be able to monitor 
progress (and that such an MIS should, iaeally, be interfaced 
with a similar MIS for the primary education sector) . 
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\/ 

Second Set 

That explicit comparisons be made between individuals, 
groups, communities, and later among, projects, regions or 
countries. 

That some sort of a formal or informal hierachical 
network be created (which, it is feared by some, will inevitably 
lead to the bureaucratization of functions and responsibilities 
of literacy practitioners) . 

That whatever else might be done with achievement data 
and information on effects, these data must be made available, 
in some form, to outsiders at upper levels of an informal 
hierarchy or of a formal bureaucracy, perhaps going right up to 
an international center in Unesco or at the World Bank. 

That both efficiency and cost-effectiveness should become 
a central concern of literacy organizers which "will not only 
produce more, it can also be expected to attract new resources 
(p. D-8) . " 

That those literacy practitioners, organizers, and 
program managers who do not play the game or those who do not 
play it adequately enough, not be rewarded by grants, or loans, 
or consultancy visits. 



\/ 

Third Set 

That, through evaluations, efficiency and effectiveness 
of the whole enterprise of providing adult literacy services 
will necessarily improve, and thus there will be an improvement 
in returns over invest?Tient . 

That there will be a homogenization of programs and that 
a generalization of content and methods across programs will 
occur. 

That local needs will be sacrificed as practitioners get 
seduced into playing the number game. 

That comparisons between programs, and rewards for the 
relatively better, might indeed reinforce existing structures of 
disparity and inequality. 
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That national and international actors will be able to 
play favorites with persons, constituencies, projects and 
countries for their own political ends. 



\/ 

Fourth Set 



That local actors will lose control to national and 
international actors making existing dependencies ever more deep 
and cruel. 

That evaluation will thus negate the whole process and 
the real meaning of development. 



It is quite clear even from a cursory view of the schema 
above that an evaluation of the EFA initiatives is not merely a 
technical matter of choosing among evaluation models, developing 
indicators and themes, designing tests and protocols, and 
developing comprehensive management information systems which, 
ideally, would accommodate both quantitative and qualitative 
data. Value conflicts are involved both in regard to the 
ideology, and the technology of information collection for 
evaluation and dissemination of evaluation results. We will now 
look at these implications as a whole and look for the reasons 
for some of the misgivings being expressed by some about 
evaluation plans in the making. 

CONCEPTION OF DEVELOPMENT IMPLIED IN EFA 

As we had indicated in the introduction to this paper, 
the evaluation implications of EFA are far-reaching and connect 
at some stage with development philosophy, educational policy 
and technical assistance strategy embedded in the EFA documents 
{1990a, 1990b). A careful reading of these documents bring up 
the following conception of development: 

Development is seen to relate to both the individual and 
the society. At the individual level, development objectives, 
therefore, must include both improved quality of life and the 
opportunity to shape and adapt to social and cultural change. 
Development must respect traditional knowledge and indigenous 
cultural heritage. It should reinforce values and concerns such 
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as self-reliance, participatory processes, concerns about 
population, war and peace, environmental protection, economic 
disparities, social justice, and tolerance of others. The EFA 
documents show optimism that such development is possible in 
today's world with the availability of life-enhancing knowledge, 
communication technology, and detente. 

Thus, the conception of development incorporated in the 
WC-EFA documents is quite progressive and comes quite close to 
the ideal of development sketched recently in the work of the 
Dag Hamirarskjold Foundation, an institution dedicated to 
liberal, humanistic and participative, human-scale development 
(Manfred et al-, 1989). 

EFA's CO. CEPTION OF EDUCATION 

EFA's conception of education is progressive as well- 
Education is accepted as a fundamental human right, and basic 
education for all is viewed as a historical moral imperative. 

Education is seen as an integrated system including FE, 
NFE and IFE. Primary education is emphasised because it is "the 
main delivery system for the basic education of children outside 
the family (p. D-6)". Th^^ central role of literacy is well 
understood in that literacy is considered "indispensable because 
literacy is a necessary skill in itself and the foundation of 
other life skills (p. D-6)." The importance of literacy is 
further reinforced through emphasis on printed knowledge which i 
indispensable in modern society and which is clearly impossible 
to share of without literacy. 

Educational objectives include empowerment, problem- 
solving, with due sensitivities towards values. At the same 
time, there is emphasis on scientific and technological literacy 
and teaching-learning of economic, social, political, analytical 
and managerial skills that are usable. At an operational level 
the objectives are quality, equity and efficiency. Disparities 
and disadvantages experienced by women, rural populations, ethni 
minorities, refugees, immigrants, and others are sought to be 
removed. Qualitative improvement in education is sought. And, 
of course, education is seen as a continuing life-long process, 

IMPLIED CONDITIONALITY IN TECHNICAL ASSISTANCE 

When it comes to the technical assistance strategy, the 
WC-EFA gets a bit hard-nosed. While WC-EFA is a declaration by 
the relatively developed part of the international community to 
assume some responsibility for bringing "education for all" to 
the relatively underdeveloped part of the world, there is an 
implied conditionality in the EFA pronouncements. The recipient 
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nations of the Third World are being asked to be accountable to 
multilateral institutions such as UNDP, UNESCO, UNICEF, and the 
World Bank by dropping their objections to testing adults and 
evaluating programs; and by showing results as emanating directly 
from the resources allocated by the developed world to •'education 
for all" initiatives in the developing world, 

CONTRADITIONS AND MISGIVINGS ABOUT EVALUATING EFA 

There is indeed something contradictory about development 
being, at the same time, a unique invention within existing 
contexts of communities and their renewing traditions and its 
evaluation using generalized indicators and norms to assist in 
comparisons across communities — and perhaps across cultures. 
There is again something strange about education being seen, on 
the one hand, as an experiment with one's identity, and, on the 
other hand, seeking to test the processes of being and becoming 
through observable, and behavioral indicators. Participatory 
strategies in planning and learning can not be assigned numerical 
values. Dialog does not lend to digital manipulations. 
Empowering is not possible within pyramids of power. 

There is nothing new about the fears and fobias that may 
have been generated among some stakeholders about EFA*s plans 
for evaluation. Battles about and around evaluation have been 
fought before, again and again, within multi-lateral, bi- 
lateral, national, subregional and community level programs and 
projects. Indeed, the battle-cries we now hear should be 
familiar to most of us. 

INFORMATION, CULTURE AND POLITICS 

A whole set of fears and misgivings arise from the nature 
of information and the uses to which it could be put. 
Evaluation, since it generates information, has important 
implications for culture and politics. Information, in its real 
essence, is "symbolic capital." Therefore, the process of 
developing information is also the process of capital formation, 
in this case, formation of symbolic capital. The modalities of 
utilization of this symbolic capital within human systems have 
consequences for all other "symbolic transformations** within 
those systems. In other words, how information is collected^ 
stored and distributed determines power relationships within 
groups, organizations, and cultures. Once information has become 
available, it can be used, or it can be abused. Within 
organizations, the processes of information collection and its 
subsequent accessibility and distribution will determine, fir^st, 
the hierarchical relationships and, then, the organizational 
climate. In societies, the processes of information development, 
and subsequently the patterns of distribution and flow of 
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information, will determine the nature of the political enonomy 
of a nation- At the international level, the export or out- 
flow of information can mean, at the least, greater 
vulnerability to the outsider, and, sometimes, outright 
dependency. 

TYRANNIES OF REPORTING AND COMPARING 

Th<?re are also fears about how evaluation will influence 
the "means and ends" calculus of programs. The fear is that 
with the need to report to those outside anc^ up, local 
practitioners will start dancing to distant drummers, losing 
touch with local needs, local objectives ^ind local solutions. 
This will inevitably lead to both the formalization and the 
homogenization of program content* The process will be 
sacrificed in favor of the product. There is also the fear 
that, in the process of measurement, qualities will become 
trivialized, and even those trivialized qualitative criteria 
will be ultimately sacrificed in favor of quantitative criteria. 
Finally, there is the position that while projects, programs and 
campaigns could be usefully evaluated in an intra-system sense, 
in terms of their own criteria and contexts, it does not make 
sense to evaluate them to make inter-system comparisons of 
achievements and results^ 



II 

SUGGESTED EVALUATION STRUCTURE AND STRATEGIES 

The position we take in this paper is that 
contradictions do exist, but then in the socio-technical world 
of evaluation planning, we should not erpect to have a reality 
without contradictions; and that we must do the best that is 
possible in this imperfect world. We should work towards a 
dialectic whereby we can, on the one hand, create a "culture of 
information" within each an^ every project, program and 
campaign, and, on the other hand, avoid formalization and 
homogenization of initiatives which would mean robbed 
opportunities and continued dependencies. 

Information collection, after all, is a process that both 
informs and jigforms. The entities on or about which information 
is being collected to inform others, are in the process, 
themselves, more or less, reformed — and sometimes transformed. 
Through a dialectic between the evaluator and the evaluand 
(meaning the entity being evaluated) , the evaluand develops 
predispositions and characteristics that make it more amenable to 
the processes of information collection and its u^e for informed 
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decision making* Such a conseqvence could do no harm to programs 
and could indeed do them a lot of good. 

A PLAN FOR AN OVERALL EVALUATION STRUCTURE FOR EFA 

The structure of an overall evaluation of the EFA's 
initiative, we suggest, should have two important aspects: 

!• META PURPOSES IN EFA'S EVALUATION 

The concrete purposes of the evaluation structure for EFA 
should be wrapped in a layer of meta purposes which should 
xnclude: 

^eta purpose, a; Developing and reinforcing a "culture of 
information ** . Literacy projects, programs and campaigns should 
be enabled to develop into ••cultures of information" so that 
"informed decision-making" becomes a norm and a habit for all 
stakeholders within the system* If projects, programs and 
campaigns delivering adult literacy do become cultures of 
information in the true sense, then there will be no need to ask 
for new or special evc^luation structures and to establish 
conditionalities, and .purely there will be no cause for fears 
and fobias, 

Meta purpose, b: Building institutional capacity > If 
literacy projects, programs and campaigns are expected to 
conduct evaluations of their program actions, there is the 
absolute necessity for the development and delivery of a program 
of institutional capacity building through training. Only then 
will the organizational actors who are expected to engage in the 
tasks of data collection, analysis, program development, and 
management, can actually undertake those tasks. Unless, 
institutional capacities are built within institutions, it will 
be wrong to expect effective progam development and meaningful 
evaluations . 

2- COMBINING INSTITUTIONAL ACCREDITATION WITH 

PROGRAM EVALUATION 

The evaluation structures of EFA should include, first, 
the evaluation of literacy institutions themselves, through 
accreditation, and, second, the evaluation of institutional 
actions and their results. Comings and Stein (1990) have indeed 
suggested that in the context of EFA, this approach may be the 
best strategy to use* Evaluation of institutions will be done in 
the context cf accreditation against a set of mutually agreed 
upon criteria* The accreditation approach will, on the one hand, 
ensure accountablity and, on the other hand, accommodate a whole 
diversity of local initiatives. Once the quality of delivery 
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systems, that is, of institutions, is ensured, there will be a 
lesser need to worry about individual achievement and performance 
that depend upon so many contextual factors. In our view, while 
this approach alone may not be sufficient, it can be most 
usefully combined with the evaluation of achievements, outputs 
and impact. 

According to Comings and Stein (1990), the original 
criteria of the Accreditation Commission (n.d.) can be adapted 
to adult literacy institutions as follows: 

Initial Criteria A^^^pted Criteria 



Mission 
Organization 
Administration 
Relations with Students 
Standards of Progress 
Financial Relations 
Student Services 
Educational Activities 
Educational Facilities 
Publications 



Management 



Student Services 
Instructional Program 
Facilities 



Instructional Staff 
Impact 



Further adaptation of these criteria in aiferent local 
settings will have to be attempted by groups of stakeholders in 
each individual case. 



HANDLING BACKGROUND ISSUES, SETTLING ANTECEDENT QUESTIONS 

It is unlikely that the agencies responsible for EFA will 
settle for anything less than "results." Administrators of 
programs in the Third World are likely to be more and less 
supportive, to take the money if there is any, and give them the 
figures* Therefore, we suggest that Third World literacy 
workers, in implementation of EFA, should make a virtue of the 
necessity of having to report results to a central point and do 
the following: 



SETTLING DEFINITIONAL QUESTIONS 

We will never be able to rise above the relativities of 
languages, cultures, social classes and social uses of literacy 
skills around the globe. We must, however, carefully develop 
situation-specific contextual definitions of literacy. Once 

11 



i O 



Bhola / Evaluation Implications of EFA 

this has been done, we should stick with such definitions both 
for curriculuin design and evaluation exercises. 

To get some ideas on developing operational definitions 
of literacy^ efforts such as the Competency Rating-Scales 
project located at the Phillip Institute of Technology in 
Australia should be reviewed (Griffin, 1990) . 

We might also consider defining literacy levels in terms 
of numbers of words learned. Of course, all languages are not 
ideographic like the Chinese language, wherein literacy levels 
are indeed defined by the number of characters learned. Yet, I 
am of the view that definitions of literacy levels based on 
numbers of words learned (from well constructed vocabulary lists 
rooted in social functions) may prove to be quite useful. Some 
empirical work on this hypothesis is worth doing. 

USING THE OPPORTUNITY TO REVIEW CURRICULA 
IN FORMAL AS WELL AS NONFORMAL EDUCATION 

The obligation to evaluate EFA initiatives should also be 
used as an opportunity to review curricula for adult literacy 
programs as well as for other nonformal education programs. Too 
often curricular plans have emerged in a haphazard manr.er without 
any preconception of ends and without any deliberation of means 
to achieve instructional ends. 

SETTLING QUESTIONS OF EQUIVALENCE 

Questions of equivalence between adult literacy programs 
on the one hand, and formal education, on the other hand, should 
be settled. Equivalence does not mean sameness. Indeed, 
"equivalent" knowledge and equivalent" skills should be taught 
to adults and children using different curricula. While some 
formalization of the adult literacy and adult education programs 
will be an unavoidable consequence of the search for equivalence, 
gains from this effort may be quite important for the 
establishment of a lifelong system of education with multiple 
points of entry and exit. 

CHOICE OF A MODEL OF EVALUATION PLANNING 

The model of evaluation planning chosen should allow the 
emergence of a culture of information within literacy 
organizations. An management information system (MIS), whether 
it is a pencil-and-paper system or a computerized network; must 
be at the core of the evaluation structure being planned. This 
model of evaluation planning should also be able to accommodate 
both rationalistic and naturalistic evaluation approaches. This 
would mean that the managonent information system (MIS) designed 
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for monitoring purposes will allow storage and retrieval of both 
quantitative and qualitative data (Bhola, 1990a; 1990b) . 

CHOICE OF INDICATORS OF DEVELOPMENT AND EDUCATION 

The set of indicators for adult literacy (and adult 
education) developed and used in the context of UNDP/UNESCO 
Experimental World Literacy Program have been reviewed and * 
discussed in Bhola (1979) and may be reviewed. As reported by 
Madeley (1990), UNDP has recently proposed that the existing 
indicators of development such as GNP shou}d be rejected in 
favour of three new indicators: (i) age expectancy at birth, 
(2) literacy, and (3) purchasing power. Any system of 
evaluation developed for programs m implementation of EFA must 
use these sets of indicators. 

SERVING DUAL PURPOSES OF DIAGNOSIS OF LEARNING 
AND TEST OF ACHIEVEMENTS 

Projects, programs and campaigns of adult literacy and 
nonformal education must learn to evaluate both for diagnosis of 
instructional successes and failures, and for testing 
achievement and impact. Data must be both qualitative and 
quantitative. All evaluation data and findings must be utilized 
by stakeholders directly involved in the program before it is 
released for processing and use at another level. 

HANDLING RELATIVITIES OF OBJECTIVES, STANDARDS AND 
RANKS IN REPORTING ACHIEVEMENT DATA 

To handle relativities of objectives, tests and 
standards, scores should be reported in three one-thirds: the 
uppo.r third, the middle third and the lower third. Such a 
mechanism will reconcile relativity with comparability. The 
reporting projects, programs or countries will be assumed to 
have agreed upon reasonable expectations of results achieveable 
in particular contexts, will be assumed to have done an 
acceptable job of testing for achievement, and then expressed 
their levels of satisfaction with results. 

TOWARD A CULTURE OF INFORMATION 

Elsewhere (Bhola 1990b) , a culture of information has 
been defined by the author to mean "an institutional culture or 
sub-culture that systematically generates, collects, stores, 
retrieves and utilizes appropriate information in all processes 
of decision-making and does so as a matter of course. This 
would indeed mean that the organization systematically collects 
and stores descriptive data generated through the very process 
of implementation of its objectives and programs. it means also 
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that such an organization would generate evaluative data through 
special evaluation studies appropriate] y designed and timed to be 
able to judge th value and worth of its various programs and 
projects-. Information within such an organization will flow back 
and forth horizontally; and will flow vertically across all 
levels, both up and down the system. Informed decision making 
will become a universally shared norrn^" 

In more concrete terms, a culture oi information, will 
require that campaigns, programs and projects collect and use 
data across the total grid below: 



THE EFA EVALUATION GRID 

TO ACTUALIZE 
A CULTURE OF INFORMATION 




MIS 


NE 


RE 


Life Expectancy 


Health 
indicators 


X 


X 




Usual 


Personal 


Comparisons 



achievement satisfaction; Correlations 
Literacy indicators Community 

perceptions 



Purchasing Power Economic X X 

indicators 



For the purposes of the grid above, individual projects, 
programs and campaigns will have to develop their own indicators 
for each of the cells. So^e of the indicators and evaluation 
objectives appropriate to literacy work have been included in the 
grid by way of examples. 
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RUDIMENTS OF A CONCRETE EVALUATION SYSTEM FOR EFA 

The ideas presented below are indeed initial and 
rudimentary for the simple reason that developing a necessary 
and sufficient system of evaluation of EFA initiatives is not a 
one-man's job. We will perhaps need a week-long meeting of the 
evaluation world's "Big Seven" or "Big Ten" to develop the 
structure and content of such a system. Then some more time 
will be needed to develop strategies for th' instalation and 
implementation of the system being designed. Finally, we will 
need to negotiate the system plans with stakeholders around the 
world through a political process that will again take a long 
period of time. In the context of this paper, we must be 
satisfied with some general principles, a few promising 
indications, and some useful outlines. 

SOME PRINCIPLES SUFFUSED IN THE SYSTEM 

1. As far as possible all evaluations should be internal 
evaluations. If at all an outsider is used, the outsider 
should be the closest to the insider within the relevant 
network of relations. 

2. Evaluations should reflect both the macro (that is the 
normative) and the micro (that is the socio-psychological) 
realities of individuals and groups involved in educational and 
social change. The micro evaluations should be done by workers 
at the grassroot level and primarily for diagnostic purpoFes. 

3. Data collected should be both necessary and sufficient. 

4. Both descriptive and evaluative data should be 
collected . 

5. The data collected should be both quantitative and 
qualitative so that the size and scope of the program as well as 
the meaning of the educational experiences of those involved in 
an educational program can be understood. 

6. The evaluation system should at the same time serve the 
facilitator, the program designer and the policy maker. 



SOME IDEAS ON A GENERAL FRAMEWORK 

The framework for data collection, storage, and flow will 
have to encompass the total system including the levels of the 
adult learner group, the supervisor, the district, the province, 
the state and the appropriate multilateral agency. Decisions 
will have to be made on what data should be collected, the 
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operations that should be performed on such data and in what form 
it should be sent, to whom. The nature and extent of aggregation 
of data at various levels will need careful attention. 

Tht! following simple chart can be used as a planning 
device at various levels of the system: 



MULT I -LEVEL .\NALYZER 
FOR DATA COLLECTION AND DATA FLOW 



Question (s) Data to be Processei? to Utilization Da':a 

to be answered collected be performed on of results flow 
for diagnostic/ or the data of evaluation 

reporting obtained 
purposes from other 

sources 



EVALUATION OF SWEDISH SCHOOLS AND ADULT EDUCATION: 
A CASE AT THE NATIONAL LEVEL 

The National Evaluation of Swedish Schools and [Formal] 
Adult Education (Sweden, 1989) reflects in practice what Bhola 
has been theoretically propounding for some years (1990a; 1990b) 
and is indeed an excellent example of how a national system of 
evaluation should be developed. This evaluation effort combines 
MIS data as well as data available from other outside national 
sources of educational, social and economic statistics; includes 
audits of schools and county educational sub-systems performed by 
stakeholders closest to the subsystem; and, finally, conducts or 
commissions various evaluation studies and research studies to 
undergird the total evaluation effort. By so doing it is able to 
study norms at the macro level as well as allow for evaluation 
studies within special conditions of schools, communities, and 
ethnic groups. It is thus able, on the one hand, to study 
processes and, on the other hand, to guage productivity and 
efficiency. 
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In this regard attention should be invited to Tuijnman 
(1987) who has listed several typical indicators relevant to the 
macro, that is, national level evaluation of formal adult 
education in Sweden and has then suggest'id a series of 
statistical operations that could be perfomed on such data, 
among them: coefficient of variation, enrolment rates, 
utilization indicator, indicator of growth, net participation 
index, pyramid index, dissimilarity index, representational 
index, selectivity index, gini coefficient, concentration index, 
measurement of qualitative variation and performance indicator. 

THINKING ABOUT THE INTERNATIONAL LEVEL 

At the international level, again, a whole set of 
statistical operations will have to be performed and a variety 
of formats for presentation of data will have to be worked cat. 
By way of example, one such tabulation format is offered below: 



AN EXAMPLE OF DATA DISPLAY 
AT NATIONAL / INTERNATIONAL LEVELS 



Country / Region / Subculture Year / Cycle. . , . 



Language Area Data Set Data Set Data Set 

ii^ii "B** '^C'* 



Percent in 
Upper one-third 

M 

F 

Percent in 
Middle one-third 

M 

F 

Percent xn 
Lowest one-third 

M 

F 
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The content of data sets A, B and C in the table above 
can be explicated as in the three tables below: 



Data Set "A" of the Main Table Above 



3-R's Health Purchasing Power 

R W A Indices Indices 



In the above table: R = Reading; W = Writing; and 
A = Arithmetic. 



Data Set "B" of the Main Table Above 



Self-Report on Satisfaction with 
X Y Z 



Data Set "C" of the Main Table Above 



Coiijnunity Expressions about 
P Q R 
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Data for the total evaluation system will have to come 
from the lower levels of the system to the upper levels of the 
system in a form that various statistical operations can be 
performed on them. By way of context, information on the length 
and duration of teaching cycles as well as costs per head on 
different items of inputs will be necessary. All this is not 
impossible to get done. 



CONCLUSION 

Both the political and professional implications of 
planning evaluations of "Education for All" initiatives have 
been discussed. The discussion has been necessarily brief, 
sketchy, and incomplete. International teams of experts, 
representing a whole range of stakeholders will need to meet for 
appropriate durations of time to invent a system to collect 
descriptive and evaluative data that are both necessary and 
sufficient. Such a plan will then have to be submitted to an 
essentially political process of negotiation before it can be 
adopted. Though somewhat sketchy and hurried, this paper may 
have at least raised some of the issues involved in evaluating 
EFA initiatives and suggested a few directions that may be open 
to us. It is our hope that this small contribution receives 
attention of those engaged in the important task of designing an 
evaluation system for EFA-related actions. 



NOTES 

1. The World Conference on Education for All (WCEFA) , was 

convened jointly by the executive heads of the United Nations 
Children's Fund, United Nations Development Programme, the 
United Nations Educational, Scientific and Cultural 
Organization, and the World Bank, and held in Jomtien, Thailand 
during March 5-9, 1990, 



ERIC 



19 

2n 



0 



Bhola / Evaluation Implications of EFA 



REFERENCES 

Accreditation Commission. Accreditation Criteria . 
Washington, D.C.: The Accreditation commission of the 
Association of Independent Colleges and Schools, n.d, 

Bhola, H.S. Evaluating Functional Literacy - (Literacy 
in Development: A Series of Training Monographs. H.S. Bhola, 
Series Editor). Teheran, Iran: International Institute for 
Adult Literacy Methods, 1979. 

Bhola, H.S. Evaluating Literacy for Development; 
Campaigns. Programs and Projects . Hamburg: Unesco Institute of 
Education, 1990a. (In Press). 

Bhola, H.S. A Model of Evaluation Planning, Evaluation 
Implementation, and Evaluation Management: Toward a "Culture of 
Information" Within Organizations. Paper prepared for the 
"Seminar on Evaluating Education Using Quantitative and 
Qualitative Approaches," Institute of International Education, 
Stockholm University, (Fall 1990), September 17, 1990b. 

Comings, John P and Stein, Sondra G. "An Evaluation 
Model for Building Adult Education Institutions." Paper 
presented at the Panel on Evaluation Implications of the World 
Conference on Education for All at Evaluation '90: Evaluation 
and the Formulation of Public Policy, American Evaluation 
Association Annual Meeting, October 18-20, 1990, Washington, 
D.C. 

Griffin, Patrick, et. al. Developing Competency Rating 
Scales in Adult Literacy: An Analytical Approach. Competency 
Rating Scales Project of the Australian Government, 1990. 

Inter-Agency Commission. Final Report; World Conference 
on Education for All: Meeting Basic Learning Needs, 5-9 March 
1990. Jomtien. Thailand . New York: Inter-'Agency Commission 
(UNDP, UNESCO, UNICEF, WORLD BANK) for the World Conference on 
Education for All, 1990a, 

Inter-Agency commission. World D eclaration on Education 
for All and Framework for Ac tion to Meet Basic Learning Ne^ds . 
(World Conference on Education for All, 5-9 March, 1990, 
Jomtien, Thailand) . New York: The Inter-Agency Commission for 
the World Conference on Education for All, 1990b. 

Lockheed, Marlaine E. "Building Educational Research and 
Assessment Capacity in Developing Countries." Paper presented 
at the Panel on Evaluation Implications of the World Conference 

20 



ERIC 



Bhoia / Evaluation Implications of EFA 



on Education for All at Evaluation '90: Evaluation and the 
Formulation of Public Policy, American Evaluation Association 
Annual Meeting, October 18-20, 1990, Washington, D.C. 

Madeley, John. Development is More than Money. 
Development and Cooperat ion. No. 4/1990, pp. 4-6. 

Manfred, Max-Neef et al. Human Scale Development: An 
option for the Future. Development Dialog ue, 1989:1, pp. 5-81. 

Swedish National Board of Education. National Evaluation 
of Swedish Schoo ls and Adult -Education- Stockholm: Swedish 
National Board of Education, 1989. 

Tuijnman, Albert. Monitoring Adult Education — With 
Special Referen ce to Sweden . Stockholm: ftockholm University, 
Institute of International Education, 198/. 



21 



ERIC 



